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MILITARY  TRANSFORMATION 

Progress  and  Challenges  for  DOD's 
Advanced  Distributed  Learning  Programs 


Why  GAO  Did  This  Study  What  GAO  Found 


The  Department  of  Defense  (DOD) 
spends  more  than  $17  billion 
annually  for  military  schools  that 
offer  nearly  30,000  military  training 
courses  to  almost  3  million  military 
personnel  and  DOD  civilians.  DOD 
is  transforming  its  forces,  including 
the  way  it  trains,  to  favor  more 
rapid  and  responsive  deployment. 
DOD’s  training  transformation 
strategy  emphasizes  the  use  of 
advanced  distributed  learning 
(ADL)  programs,  such  as  Internet- 
based  training,  as  critical  to 
achieving  its  training  and 
overarching  transformation  goals. 
ADL  is  instruction  that  does  not 
require  an  instructor’s  presence; 
can  use  more  than  one  media;  and 
emphasizes  the  use  of  reusable 
content,  networks,  and  learning 
management  systems. 

Because  of  ADL’s  importance  to 
DOD’s  transformation  efforts  and 
pursuant  to  GAO’s  basic  legislative 
responsibilities,  we  initiated  this 
review  to  create  a  baseline 
document  that  describes  the  status 
of  DOD’s  ADL  programs.  GAO 
reviewed  these  programs  to 
determine  (1)  DOD’s  expectations 
for  the  programs;  (2)  the 
implementation  status  of  those 
programs;  and  (3)  major  challenges 
affecting  program  implementation. 
GAO  did  not  assess  the  programs’ 
effectiveness  at  this  time  because 
most  are  in  the  early  stages  of 
implementation. 

DOD  reviewed  a  draft  of  this  report 
and  concurred  with  its  contents. 


DOD  has  set  high  expectations  for  ADL.  They  expect  the  programs  to  provide 
new  learning  opportunities  and  technologies  across  a  wide  range  of  training 
areas.  Ultimately,  a  key  benefit  of  ADL  is  expected  to  be  improved  readiness 
through  reengineering  of  training  and  enhancing  service  members’  skills. 

DOD,  the  services,  and  Joint  Staff  are  generally  in  the  early  stages  of 
implementing  their  ADL  programs  and  have  made  progress  in  several  areas. 
OSD,  with  its  three  ADL  co-laboratories;  the  services;  and  the  Joint  Staff  chose 
an  industry-wide  ADL  standard  for  content  interoperability  and  collaboration 
across  the  services.  They  promoted  experimentation  with  new  technology  and 
working  with  private  industry.  The  services’  programs  generally  focus  on 
distribution  infrastructure  and  service-specific  content  development.  According 
to  ADL  program  officials,  OSD,  the  Joint  Staff,  and  the  services  have  achieved 
some  ADL  successes.  For  example,  OSD,  in  collaboration  with  the  co¬ 
laboratories,  developed  successful  course  content  prototypes;  and  the  Army’s 
Battle  Staff  Noncommissioned  Officer  course  resulted  in  annual  savings  while 
maintaining  student  performance.  However,  it  is  too  early  to  fully  assess  the 
extent  of  each  program’s  effectiveness. 

DOD  faces  cultural,  technological,  policy  and  financial  challenges  that  affect  the 
ADL  programs’  ability  to  fully  achieve  the  benefits  of  enhanced  learning  and 
performance  and  of  improved  readiness.  Key  challenges  are  summarized  below. 


Challenges  Affecting  DOD’s  ADL  Programs 


Challenge 

Description 

Cultural 

• 

• 

• 

Crganizational  culture  is  resistant  to  change. 

Senior  leadership  commitment  varies:  preference  is  for  the  more 
traditional  schoolhouse-focused  learning. 

Service  schoolhouses  are  reluctant  to  change  since  funding  and 
infrastructure  are  closely  tied  to  numbers  of  in-resident  students. 

Technological 

• 

• 

Bandwidth  issues  and  unresolved  network  security  concerns  stifle  utility. 

The  development,  fielding  of,  and  access  to  military  skills-related  content 
is  more  difficult  and  costly  than  anticipated. 

Policy 

• 

OSD  is  in  the  early  stage  of  formulating  policy  that  specifically  addresses 
the  use  of  ADL,  consequently  some  of  the  military  services'  and 's  training 
and  education  regulations  are  outdated  and  awaiting  a  definitive  policy. 

Financial 

• 

• 

• 

Budget  and  funding  issues  for  the  long-term  use  of  ADL  are  unresolved. 
Allocated  funds — around  $431  million,  less  than  1.3  percent  of  its  training 
budget  for  fiscal  years  1 999  through  2002 — did  not  always  meet 
requirements. 

Projected  program  needs — $2.2  billion  for  fiscal  years  2003  through 

2007 — is  about  $600  million  more  than  currently  programmed. 

Source:  DOD. 

Note:  GAO  analysis  of  OSD,  Joint  Staff  and  service  data. 

www.gao.gov/cgi-bin/getrpt7GAO-03-393. 

To  view  the  full  report,  including  the  scope 
and  methodology,  click  on  the  link  above. 

For  more  information,  contact  Neal  P.  Curtin, 
at  (757)  552-8100  or  e-mail  curtinn@gao.gov. 
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DUSD®  Deputy  Under  Secretary  of  Defense  for  Readiness 

ECP  Extension  Course  Program 

ETSC  Education  and  Training  Steering  Committee 

IRR  Individual  Ready  Reserve 

ITV  Interactive  Television 

JCLE  Joint  Collaborative  Learning  Environment 

JPME  II  Joint  Professional  Military  Education  II 

LMS  Learning  Management  System 

MC  Marine  Corps 

MCDLP  Marien  Corps  Distance  Learning  Program 
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NIPRNET  Non-Classified  Internet  Protocol  Router  Network 
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OSD  Office  of  the  Secretary  of  Defense 

PME  Professional  Military  Education 
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QDR  Quadrennial  Defense  Review 

R,D,T&E  Research,  Development,  Test  and  Evaluation 

SCORM  Sharable  Content  Object  Reference  Model 

SIPRNET  Secret  Internet  Protocol  Router  Network 

TADLP  The  Army  Distributed  Learning  Program 

TFADLAT  Total  force  Advanced  Distributed  Learning  Action  Team 

TRADOC  U.S.  Army  Training  and  Doctrine  Command 

VTC  Video  Teleconference 

VTT  Video  Teletraining 


This  is  a  work  of  the  U.S.  Government  and  is  not  subject  to  copyright  protection  in  the 
United  States.  It  may  be  reproduced  and  distributed  in  its  entirety  without  further 
permission  from  GAO.  It  may  contain  copyrighted  graphics,  images  or  other  materiais. 
Permission  from  the  copyright  hoider  may  be  necessary  shouid  you  wish  to  reproduce 
copyrighted  materials  separately  from  GAO’s  product. 
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i 

GAO 

^^^^^^^^^Accountability  *  Integrity  *  Reliability _ 

United  States  General  Acconnting  Office 
Washington,  DC  20548 


February  28,  2003 

The  Honorable  John  Ensign 
Chairman 

The  Honorable  Daniel  K.  Akaka 
Ranking  Minority  Member 

Subcommittee  on  Readiness  and  Management  Support 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Joel  Hefley 
Chairman 

The  Honorable  Solomon  P.  Ortiz 
Ranking  Minority  Member 
Subcommittee  on  Readiness 
Committee  on  Armed  Services 
House  of  Representatives 

The  Department  of  Defense  (DOD)  spends  more  than  $17  billion^  annually 
for  military  schools  that  offer  nearly  30,000  military  training  courses  to 
almost  3  million  military  personnel  and  DOD  civilians,  much  of  it  to 
maintain  readiness.  ^  To  better  meet  the  diverse  defense  challenges  of  the 
future,  DOD  is  transforming  its  forces,  including  its  training,  for  a  post- 
Cold  War  environment  that  favors  more  rapid  deployment  and 
responsiveness.  DOD’s  Training  Transformation  Strategy*  emphasizes  the 
use  of  advanced  distributed  learning  (ADL)  programs  such  as  Internet- 
based  training,  as  critical  to  achieving  the  department’s  training  and 
overarching  transformation  goals  and  to  deliver  the  highest  quality  training 
cost-effectively  anytime,  anywhere,  whether  active  duty,  reserve,  or 
civilian  personnel.  ADL  is  instruction  that  does  not  require  an  instructor’s 


*This  amount  includes  the  cost  of  conducting  school  training,  including  instructor’s  pay; 
classroom  availability  and  operation;  course  development;  and  student’s  military  pay, 
billeting  cost,  and  temporary  duty  costs. 

^Generally,  formal  military  training  and  education  occurs  at  centralized  training  facilities 
and  lasts  weeks  or  months. 

**Office  of  the  Under  Secretary  of  Defense  for  Personnel  and  Readiness,  Strategic  Plan  for 
Transforming  DOD  Training,  March  1,  2002.  In  this  plan  the  definition  of  “training”  is 
expanded  to  include  training,  education,  and  job  performance  aiding.  OSD’s  training 
transformation  implementation  plan  should  be  completed  by  March  2003. 
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presence;  can  use  more  than  one  media;  and  emphasizes  the  use  of 
reusable  content,  networks,  and  learning  management  systems/ 

We  initiated  this  review  of  DOD’s  ADL  programs,  pursuant  to  our  basic 
legislative  responsibilities,  because  of  the  importance  DOD  has  placed  on 
them  as  a  key  to  achieving  the  department’s  transformation  efforts. 
Specifically,  we  addressed  the  following  questions:  (1)  What  are  DOD’s 
expectations  for  the  programs?  (2)  How  is  DOD  managing  ADL  and  what 
progress  is  being  made  in  implementing  the  programs?  (3)  What  major 
challenges  are  affecting  the  programs’  implementation?  We  did  not  assess 
the  effectiveness  of  the  programs  at  this  time  because  most  are  in  the  early 
stages  of  implementation;  thus  our  objective  was  to  provide  a  baseline 
document  concerning  the  focus,  status,  and  magnitude  of  DOD’s  ADL 
programs. 

In  late  August  and  early  September  2002,  because  of  your  continuing 
interest  in  the  readiness  and  training  of  U.S.  armed  forces,  we  briefed  your 
offices  and  those  of  Representatives  John  McHugh  and  Adam  Smith  on  the 
results  of  our  work.  This  report  summarizes  and  updates  the  major 
observations  provided  at  our  briefings.  (See  briefing  sections  I  through  X.) 

We  conducted  our  review  from  February  2002  through  August  2002  in 
accordance  with  generally  accepted  government  auditing  standards. 
Appendix  I  describes  our  scope  and  methodology. 


Background 


The  increased  rate  of  deployments  in  recent  years  of  DOD’s  forces,  which 
often  involve  rapid,  unplanned  movements  to  locations  around  the  world, 
highlights  the  need  for  the  services  to  provide  training  on  demand  to 
soldiers  and  units  deployed  worldwide.  Accordingly,  because  of  more 
demanding  deployment  criteria  and  other  time-sensitive  constraints,  DOD 
recognized  that  yesterday’s  framework  “right  time,  right  place”  learning, 
with  its  use  of  set  times  and  places  for  training,  may  not  meet  future 
military  requirements.  It  also  recognizes  that  providing  “anytime, 
anywhere”  instruction  is  essential  to  maintaining  military  readiness  in  the 
information  age,  where  future  forces  and  their  support  activities  need  to 
be  highly  adaptive  to  meet  threats  effectively  and  rapidly. 


'‘Reusable  content  includes,  but  is  not  limited  to,  courseware,  tutorials,  and  case  studies; 
networks  are  Intra-  or  Internet  based;  and  learning  management  systems  are  operating 
systems  that  provide  access  to  “content  objects”  and  help  register,  track,  and  administer 
courses  to  a  given  student  population. 
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In  response  to  the  DOD  1997  Quadrennial  Defense  Review/  the 
department  developed  a  DOD-wide  strategy  to  use  learning  and 
information  technologies  to  modernize  education  and  training.  The  initial 
effort  in  that  development  was  the  ADL  Initiative.  Its  intent  was  to  set 
forth  a  new  framework  to  provide  DOD  personnel  access  to  high  quality 
education  and  training,  tailored  to  individual  needs  and  delivered  cost- 
effectively,  whenever  and  wherever  it  is  required.  DOD  envisioned  using 
the  Internet  and  other  virtual  or  private  wide-area  networks,  distributed 
learning  experts,  learning  management,  and  diverse  support  tools  to 
ensure  a  “learner-centric”  ADL  system  that  delivers  high  quality  training, 
education,  and  job  performance  aiding.  DOD  sees  ADL  programs  as  part  of 
a  continuum®  of  learning  that  encompasses  many  learning  methodologies, 
as  shown  in  table  1. 


^William  S.  Cohen,  Secretary  of  Defense,  Report  of  the  Quadrennial  Defense  Review,  May 
1997. 

®A  continuum  is  defined  as  a  whole  characterized  as  a  collection,  sequence,  or  progression 
of  elements  varying  by  minute  degrees. 
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Table  1:  Continuum  of  Learning  Methods 


Right  time,  right  place 

Anytime,  anywhere 

Classroom 

Distance/distributed 

Advanced  distributed 

delivery  method 

learning  delivery  methods 

learning  delivery  methods 

•Instructor-led 

•Video  tele-training 

•Integrated  networked  systems 

training 

•Embedded  training 

•Integrated  platforms 

•Computer  conferencing 

•Reusable  learning  objects 

•Interactive  television 

•Widespread  collaboration 

•Electronic  classrooms 

•Global  knowledge  databases 

•Interactive  multimedia 

•Intelligent  tutoring  systems 

•Computer-based  training 

•Performance  aiding 

•Audio-graphics 

•Digital  knowledge  repositories 

•Audiotapes/videotapes 

•Internet-based  instruction 

•Correspondence  courses 

•Virtual  libraries 

•Simulations 

•Virtual  classrooms 

Source:  Defense  Acquisition  University. 


Note:  The  data  displayed  in  the  table  is  based  on  data  provided  in  the  Defense  Acquisition 
University’s  Strategic  Plan  2002-2009  Training  Transformation  (T2),  The  DAU  Road  Map  for  e- 
Learning  and  On-line  Performance  Support. 


In  April  1999,  DOD  issued  its  ADL  strategy^  in  response  to  the  1997  DOD 
Quadrennial  Defense  Review.  The  strategy  also  responded  to  (1)  the 
directive  in  the  National  Defense  Authorization  Act  for  Fiscal  Year  1999® 
for  DOD  to  develop  a  strategic  plan  to  guide  and  expand  distributed 
learning  initiatives  and  (2)  Executive  Order  13,111®  that  tasked  DOD  to 
provide  guidance  to  Defense  agencies  and  advise  civilian  agencies  in 
developing  and  implementing  collaborative  distance  learning  standards. 
DOD’s  strategic  plan  defined  ADL  as  a  way  to  leverage  the  power  of 
computer,  information,  and  communication  technologies  through  the  use 
of  common  standards  in  order  to  provide  learning  that  can  be  tailored  to 
individual  needs  and  delivered  anytime,  anywhere,  in  either  training  or 
education  environments.  It  also  includes  establishing  an  interoperable 
“computer-managed  instruction”  environment  to  support  the  needs  of 
developers,  learners,  instructors,  administrators,  managers,  and  family.  An 


^Office  of  the  Under  Secretary  of  Defense  for  Personnel  and  Readiness,  Report  to  the  106th 
Congress,  Department  of  Defense  Strategic  Plan  for  Advanced  Distributed  Learning,  Apr. 
30,  1999. 

^Public  Law  105-261,  sec.  378,  Oct.  17,  1998. 

®Exec.  Order  13,111,  Using  Technology  to  Improve  Training  Opportunities  for  Federal 
Government  Employees,  sec.  4  (c),  Jan.  12,  1999. 
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ADL  implementation  plan  followed  in  May  2000  to  provide  a  federal 
framework.  It  described  the  department’s  approach  to  carrying  out  its 
strategic  plan  and  provided  an  update  on  each  of  the  services’  and  the 
Joint  Staffs  programs. Since  1995,  OSD,  the  services,  and  the  Joint  Staff 
have  established  ADL  programs  in  concert  with  key  executive, 
congressional,  and  departmental  guidance  discussed  above.  See  appendix 
11  for  a  timeline  of  key  events. 

OSD’s  March  2002  Training  Transformation  Strategy  emphasizes  the  use  of 
ADL  programs  as  critical  to  achieving  the  department’s  training  and 
overarching  transformation  goals  and  ensuring  that  training  is  readily 
available  to  both  active  and  reserve  military  personnel,  regardless  of  time 
and  place.  The  training  transformation  strategy  and  soon  to  be  released 
implementation  plan  are  intended  to  reengineer  training;  enhance  service 
members’  skills;  and  provide  capabilities-based  training  to  support  service, 
joint,  interagency,  intergovernmental,  and  multinational  operations. 


Summary 


Officials  from  OSD,  the  services,  and  the  Joint  Staff  have  set  high 
expectations  for  ADL.  They  expect  the  programs,  which  include  the 
various  delivery  methods  cited  in  table  1,  to  provide  new  learning 
opportunities  and  technologies  and  improved  readiness.  In  terms  of  new 
learning  opportimities  and  technologies,  DOD  expects 

•  increased  accessibility  to  training  for  personnel, 

•  interoperability  of  instruction  components  in  varied  locations  by 
different  services, 

•  reusability  in  multiple  applications, 

•  durability,  despite  changes  in  technology,  and 

•  affordability. 


^"office  of  the  Deputy  Under  Secretary  of  Defense  (Readiness),  Director  for  Readiness  and 
Training,  Department  of  Defense  Implementation  Plan  for  Advanced  Distributed 
Learning,  May  19,  2000. 
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With  regard  to  improved  readiness,  DOD  expects  ADL  to  improve 
readiness  by 

•  supporting  the  training  transformation  initiative  and  the  combatant 
commanders, 

•  enhancing  training  opportunities  for  joint  assignments, 

•  enhancing  training  opportunities  for  reserve  personnel, 

•  improving  mission  performance  through  anytime,  anywhere,  and  just- 
in-time  assignment-oriented  and  job  performance  enhancement 
training,  and 

•  improving  maiming  by  reducing  personnel’s  nonavailability  and  unit 
turbulence  and  reducing  time  for  in-resident  training  with  large  return- 
on-investment  for  temporary  duty  costs,  while  increasing  retention  and 
quality-of-life  enrichment.  (See  briefing  section  11.) 

OSD,  the  services,  and  the  Joint  Staff  are  generally  in  the  early  stages  of 
implementing  their  ADL  programs  and  have  made  progress  in  several 
areas.  OSD’s  Office  of  the  Deputy  Under  Secretary  of  Defense  for 
Readiness  provides  executive  policy  and  programmatic  oversight  and 
guidance  for  the  department’s  ADL  implementation.  That  office  also  leads 
a  collaborative  effort  to  produce  ADL  policy,  plans,  and  procedures  for 
developing  and  implementing  ADL  technologies  across  the  department. 
This  collaboration  involves  the  services.  Joint  Staff,  other  DOD 
components,  the  ADL  collaborative  laboratories  (co-labs),  the  Coast 
Guard,  and  the  Department  of  Labor.  For  example,  OSD  in  collaboration 
with  its  partners,  chose  an  industry-wide  ADL  standard  for  content 
interoperability  to  be  used  throughout  DOD,  which  allows  for 
collaboration  of  course  content  across  the  services.  The  standard, 

Sharable  Content  Object  Reference  Model  (SCORM),  is  an  evolving  set  of 
technical  specifications  designed  to  ensure  the  interoperability, 
accessibility,  and  reusability  of  on-line  courseware.  The  Joint  Staff  and  the 
services  agree  that  future  course  content  will  be  designed  to  conform  to 
SCORM.  OSD,  with  the  National  Guard  Bureau  (NGB)  and  the  Department 
of  Labor,  established  three  ADL  co-labs  to  experiment  with  new 
technology  and  leverage  experience  between  private  industry  and  military 
components.  It  also  participates  in  an  international  partnership  co-lab  in 
Telford,  England,  to  promote  collaboration  and  global  e-learning.  The 
services’  and  Joint  Staffs  programs — individual  programs  designed  by  and 
tailored  for  the  specific  needs  of  each  service  or  joint  position — share  a 
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similar  vision  of  providing  learner-centric  (i.e.,  on  demand,  “anytime, 
anywhere”)  training  and  focus  on,  among  other  aspects,  distribution 
infrastructure  and  service-specific  content  development.  (See  briefing 
section  Ill.) 

OSD,  the  Joint  Staff,  and  the  services  note  that  they  have  achieved  some 
ADL  successes,  such  as  the  following: 

•  OSD — with  the  co-labs,  military  services.  Joint  Staff,  and  co-sponsors — 
developed  successful  content  prototypes,  including  one  joint 
professional  military  education  course." 

•  The  Joint  Staffs  Joint  Collaborative  Learning  Environment  prototype 
established  an  initial  joint  personnel  tracking  and  portal  capability. 

•  The  Army’s  Battle  Staff  Noncommissioned  Officer  course  conversion  to 
an  ADL  format  resulted  in  a  $2.9  million  annual  cost  avoidance  while 
maintaining  student  performance. 

•  The  Navy — to  promote  interoperability,  ease  of  access  to  DOD  Internet 
sites,  and  reduce  training  time — established  both  .mil  and  .com  access 
to  ADL  courses. 

•  The  Marine  Corps’  distance  learning  application  in  terrorism  awareness 
reduced  training  time  from  11  hours  to  6  hours  and  increased  the 
average  exam  scores  by  7  percentage  points. 

•  The  Air  Force  developed  CD-ROM  training  for  hazardous  material 
incident  response  for  DOD  firefighters  and  law  enforcement  personnel 
that  reportedly  resulted  in  a  significant  increase  of  certified  responders 
and  a  projected  $16.6  million  cost  avoidance. 

Additionally,  the  Defense  Acquisition  University’s  (DAU)  ADL  program  is 
cited  by  DOD  ADL  program  officials  as  a  success  and  an  example  of  “best 
practices.”  According  to  university  officials,  since  1998  on-line 


"joint  professional  military  education  is  a  Joint  Chief  of  Staff-approved  body  of  objectives, 
policies,  procedures,  and  standards  supporting  the  educational  requirements  for  joint 
officer  development. 

Defense  Acquisition  University,  the  “corporate  university”  for  DOD,  provides  the 
acquisition,  technology,  and  logistics  community  with  learning  products  and  services.  Its 
distance  learning  program  currently  provides  19  on-line  courses. 
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instructional  time  increased  from  15,750  hours  to  1.4  million  hours; 
graduates  attending  on-line  training  courses  increased  38  percent;  and  the 
on-line  program  management  curriculum  reduced  armual  student  training 
weeks  from  36, 120  to  10,000 — a  real  savings  of  300  annual  work  years  or 
$17.4  million.  The  university’s  program  was  awarded  the  U.S.  Distance 
Learning  Association  Award  for  Excellence  in  Government  in  2001  and 
2002  for  the  quality  of  its  on-line  offerings. 

A  number  of  cultural,  technological,  policy,  and  financial  challenges  affect 
OSD’s,  the  services’  and  the  Joint  Staffs  ability  to  execute  programs  that 
achieve  the  attainable  benefits  of  enhanced  learning  and  performance  and 
improved  readiness  in  concert  with  DOD’s  ADL  vision  and  training 
transformation  strategy.  According  to  DOD  officials,  there  is  a  strong 
interrelationship  among  the  challenges  and  that  a  solution  for  one 
challenge  may  have  an  impact  on  the  others. 

Cultural  A  major  cultural  barrier,  according  to  DOD  ADL  program  officials,  is  the 

varying  level  of  commitment  of  senior  military  and  civilian  leadership  in 
the  military.  The  consensus  view  of  the  ADL  program  officials  we 
contacted  was  that  not  all  senior  military  and  civilian  leadership  is 
committed  to  ADL,  preferring  the  traditional,  schoolhouse-focused 
approach  to  learning.  Hesitance  to  embrace  ADL  is  also  explained  as  a 
function  of  less  familiarity  and  comfort  by  senior  officials  with  computers, 
advanced  technologies,  and  emerging  policies.  Similarly,  ADL  program 
officials  told  us  that  the  military  services’  schoolhouses  are  reluctant  to 
change,  in  large  part  because  their  funding  and  infrastructure  are  tied  so 
closely  to  the  number  of  students  actually  trained  on-site. 

Technological  According  to  DOD  officials,  the  services  are  all  moving  toward  Web-  or 

Internet-based  access  to  course  content  in  support  of  DOD’s  vision  of 
“anytime,  anywhere”  delivery  of  training.  The  officials  stated  that  much 
progress  has  been  made  to  enable  this  type  of  access.  However,  according 
to  OSD  and  service  officials,  bandwidth  is  generally  insufficient  to  support 
interactive,  multimedia  learning  content  and  simulations;  and  unresolved 
network  security  concerns  stifle  utility.  For  example,  we  recently  reported 
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that  the  National  Guard  Bureau  cannot  ensure  that  GuardNet^  will 
perform  as  intended  or  provide  its  users  with  reliable  and  secure  services 
because  the  requirements,  corrfiguration,  and  security  processes  for 
managing  the  network  are  ineffective.  “  DOD  ADL  officials  acknowledge 
the  same  issue  exists  throughout  DOD.  Perhaps  more  significantly,  the 
development  of,  fielding  of,  and  access^  to  military  skills-related  course 
content  that  could  most  positively  impact  readiness  continue  to  be  more 
difficult  than  anticipated,  leading  to  higher  costs  and  slower  content 
availability  than  forecasted. 

Policy  Some  of  DOD’s  training  policies  are  obsolete;  consequently,  some  of  the 

military  services’  training  regulations  do  not  reflect  the  availability  or  use 
of  new  ADL  technologies.  “  For  example,  according  to  DOD  officials,  DOD 
is  in  the  early  stage  of  formulating  policy  that  specifically  addresses  the 
use  of  ADL.  DOD  officials  believe  that  without  an  OSD-specific  ADL 
policy,  many  of  DOD’s  policies  and  guidance  documents  will  require 
updating,  so  as  to  provide  a  requirement  for  the  military  service’s  in  turn, 
to  update  their  training  and  education  regulations  that  address  the  use  of 
ADL.  Also,  the  Army’s  primary  training  regulations^  has  been  awaiting 


s^National  Guard  Bureau’s  GuardNet,  the  NGB’s  wide-area  network,  was  initially 
established  to  support  Web-based  distance  learning  for  its  units  in  the  states,  the  U.S. 
territories,  and  the  District  of  Columbia.  GuardNet,  a  network  of  interconnected  federal 
and  state  mihtary  networks  across  the  United  States,  can  connect  to  a  defense  network 
operated  by  the  Defense  Information  Systems  Agency,  and  through  this  network  to  the 
Internet.  GuardNet  has  recently  been  used  to  support  homeland  security  activities  such  as 
emergency  command  and  control  functions,  airport  security  activities  coordination,  and 
public  service  announcements. 

^''U.S.  General  Accounting  Office,  National  Guard:  Effective  Management  Processes 
Needed  for  Wide-Area  Network,  GAO-02-959  (Washington,  D.C.:  Sept.  24,  2002). 

^^For  this  report,  access  refers  to  the  availabihty  and  ability  to  access  computer  hardware, 
sufficient  bandwidth  to  support  multimedia,  interactive  course  content,  and/or  available 
duty  time  to  accomplish  ADL. 

^^DOD  policies  and  regulations  include,  but  may  not  be  limited  to.  Department  of  Defense 
Directive  1200.16,  Contracted  Civilian-Acquired  Training  (CCAT)  for  Reserve 
Components,  May  30,1990;  Department  of  Defense  Directive  1322.18,  Military  Training, 
Jan.  9,  1987;  Department  of  Defense  Directive  1430.13,  Training  Simulators  and  Devices, 
Aug.  22,  1986;  Department  of  Defense  Directive  8320.1,  DOD  Data  Administration,  Sept. 
26,  1991;  Department  of  Defense  Directive  8000.1,  Management  of  DOD  Information 
Resources  and  Information  Technology,  Feb.  27,  2002;  and  Department  of  Defense 
Instruction  5200.40,  DOD  Information  Technology  Security  Certification  and 
Accreditation  Process,  Dec.  30,  1997. 

^^Department  of  the  Army,  Army  Regulation  350-1,  Army  Training,  Aug.  1,  1983. 
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revision  for  3  years,  in  part,  due  to  a  lack  of  consensus  on  integrating  new 
technologies,  including  ADL,  with  traditional  training  approaches. 

Financial  Funding  and  budgeting  issues  similar  to  those  we  reported  for  DOD’s 

distance  learning  programs  in  1997  remain  unresolved.^®  Funding 
allocations  of  more  than  $431  million  for  fiscal  years  1999  through  2002 
(less  than  1.3  percent  of  its  training  budget  during  that  period)  did  not 
always  meet  program  requirements,  which  were  difficult  to  determine  for 
a  new  program  where  standards  were  evolving  and  the  technology 
changing  rapidly.  It  is  not  likely  that  planned  funding  levels  will  meet 
future  expected  requirements.  DOD  program  officials  project  that  over 
$2.2  billion  will  be  needed  for  ADL  programs  through  fiscal  year  2007  but 
currently  have  programmed  about  $1.6  billion — a  more  than  $600  million 
funding  gap.  Furthermore,  according  to  DOD  program  officials,  in  some 
cases,  anticipated  training  savings  attributable  to  ADL  implementation 
were  removed  from  the  budget  as  savings  before  they  were  realized. 
According  to  service  officials,  some  training  facility  commanders  continue 
to  be  concerned  that  ADL  will  reduce  their  resources  because  of  the 
decrease  in  the  number  of  students  receiving  traditional  schoolhouse 
training.  Finally,  the  Joint  Staff  and  the  services  are  still  considering  how 
to  budget  for  the  long-term  use  of  ADL.  (See  briefing  section  IV.) 


Agency  Comments 


The  Deputy  Under  Secretary  of  Defense  (Readiness)  provided  written 
comments  on  a  draft  of  this  report,  which  are  reprinted  in  their  entirety  in 
appendix  111.  In  its  comments,  DOD  concurred  with  the  content  of  the 
report.  DOD  also  provided  technical  comments  to  the  draft,  which  we 
have  incorporated  as  appropriate. 


^®U.S.  General  Accounting  Office,  Distance  Learning:  Opportunities  Exist  for  DOD  to 
Capitalize  on  Services’  Efforts,  GAO/NSIAD-98-63R  (Washington,  D.C.:  Dec.  18,  1997).  We 
reported  that  the  resolution  of  funding  and  budgeting  issues  would  benefit  the  services’ 
distance  learning  initiatives.  These  issues  are  the  (1)  extent  of  investment  that  will  be 
needed  to  convert  selected  courses  and  delivery  infrastructures;  (2)  dollar  savings  that  can 
be  realized;  (3)  impact  on  the  current  training  infrastructure,  in  terms  of  requirements  for 
instructors,  training  developers,  training  equipment,  course  maintenance,  and  training 
facility  operations;  and  (4)  process  for  budgeting  for  long-term  use  of  distance  learning. 
Distance  learning  is  structured  training  that  can  take  place  almost  anywhere  and  anytime 
without  the  physical  presence  of  an  instructor  and  may  use  one  or  more  media  but,  unhke 
ADL,  does  not  emphasize  the  use  of  reusable  objects,  networks,  and  learning  management 
systems. 
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We  are  sending  copies  of  this  report  to  Representatives  John  McHugh  and 
Adam  Smith  and  other  congressional  members  as  appropriate.  We  will 
also  send  copies  to  the  Secretary  of  Defense;  the  Secretaries  of  the  Army, 
the  Navy,  and  the  Air  Force;  and  the  Commandant  of  the  Marine  Corps. 

We  will  make  copies  available  to  others  on  request.  In  addition,  the  report 
will  be  available  at  no  charge  on  the  GAO  Web  site  at  http://www.gao.gov . 

If  you  or  yom  staff  have  any  questions,  please  call  me  on  (757)  552-8100  or 
Clifton  Spruill,  Assistant  Director,  on  (202)  512-4531.  Major  contributors  to 
this  report  were  Claudia  Dickey,  Arnett  Sanders,  James  Walker,  M.  Jane 
Hunt,  Susan  Woodward,  and  Scott  Garmon. 


Neal  P.  Curtin 

Director,  Defense  Capabilities 
and  Management 
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Briefing  Section  I:  Background  on  DOD’s 
Advanced  Distributed  Learning  Programs 


^GAO 

Accountability  *  Irtegrilv  *  Rgltabllity 

DOD’s  ADL  Programs 
Requirements  and  Directives 

•  1997  Quadrennial  Defense  Review  (QDR)  -  Directed  DOD- 
wide  strategy  to  use  learning  and  information  technologies  to 
modernize  education  and  training. 

•  1 999  Defense  Authorization  Act  -  Directed  DOD  to  develop  a 
strategic  plan  to  guide  and  expand  distributed  learning 
initiative. 

•  1 999  Executive  Order  1 3.  Ill  -  Tasked  DOD  to  lead  federal 
agencies  in  developing  collaborative  standards. 

•  2001  QDR  -  Recognized  training  transformation  as  key  to 
achieving  DOD-wide  transformation  goals. 

•  FY  2003-2007  Defense  Planning  Guidance  -  Directed  OSD  to 
develop  a  DOD-wide  training  transformation  strategy  to 
ensure  distributed  learning  technologies  are  used  to 
reengineer  training  and  enhance  service  members’  skills. 
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Briefing  Section  I:  Background  on  DOD’s 
Advanced  Distributed  Learning  Programs 


^^GlAO_  Background 

Accountafailrty  *  Integrity  *  Relwbllity 


Table  2:  Potential  Students  within  DOD  by  Component 


Active 

Officer 

Enlisted 

Total  ft 

Army 

76,067 

401,138 

477,205  1 

Air  Force 

69,466 

288,720 

358,186  1 

Navy 

53,972 

323,745 

377,717  1 

Marine  Corps 

18,393 

154,348 

172,741  1 

Total  Acti\e 

1,385,849  1 

Reserves  | 

Army 

41,908 

171,961 

213,869 

Air  Force 

17,730 

60,043 

77,773 

Navy 

19,660 

73,464 

93,124 

Marine  Corps 

4,026 

37,441 

41,467 

Army  National  Guard 

36,056 

316,382 

352,438 

Air  National  Guard 

13,790 

98,123 

111,913 

Total  Reserve 

890,584 

Civilians 

Army 

228,987 

Air  Force 

150,940 

Navy/Marine  Corps 

181,972 

OSD  and  other 

Defense  Agencies 

102,405 

Total  Civilians 

664,304 

Grand  Total 

2,940,737 

Source:  Washington  Headquarters  Service  Directorate  for  Information  Operations  and  Reports  and  Defense  Manpower  Data  Center. 

Notes:  Data  is  as  of  Apr.  2002. 

Reserve  Component  numbers  include  Selective  Reserve  Personnel, 
Individual  Ready  Reserve  (IRR)  and  Standby  Reserve  personnel. 
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Briefing  Section  II:  DOD’s  Expectations  for 
Advanced  Distributed  Learning  Programs 


^  G  A  Q  DOD’s  Expectations  for  ADL 

AccountaPilfty  *  Integrity  *  Reliaftllity  I 

ADL  is  Expected  to  Improve 
Learning  and  Readiness 

•  OSD  expects  ADL  implementation  to  provide  a 
continuum  of  life-long  learning  environment  that 
includes 

•  improved  learning  opportunities  and  new  learning 
technologies  and 

•  improved  readiness  to  support  the  combatant 
commanders. 

•  The  services  and  Joint  Staff  expect  their  programs 
to  provide  similar  learning  opportunities  and 
technology  improvements  but  are  more  focused  on 
improved  performance  and  readiness. 
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Briefing  Section  II:  DOD’s  Expectations  for 
Advanced  Distributed  Learning  Programs 


^  G  AO  DOD’s  Expectations  for  ADL 

Accountability  *  Integrity  *  Reliability  I 

ADL  Has  the  Potential  to  Provide  Improved 
Learning  Opportunities  and  Learning  Technologies 

ADL  has  the  potential  to  provide  improved  learning  opportunities  and 

technologies  that  may  result  in  improved: 

•Accessibility 

•  Ability  to  access  instruction  from  one  remote  location  and  deliver  it  to  many 
other  locations,  and  training  and  technology  reach-back  for  deployed 
soldiers. 

•Interoperability 

•  Ability  to  use  instruction  components  developed  in  one  location  in  another 
location,  or  by  more  than  one  service. 

•Reusability 

•  Ability  to  design  instruction  components  that  can  be  incorporated  into 
multiple  applications. 

•Durability 

•  Ability  to  continue  using  instruction  components  without  redesign  or  recoding 
when  base  technology  changes. 

•Affordability 

•  Ability  to  reduce  instruction  time,  number  of  students  per  course  at  a 
resident  schoolhouse  and,  education  infrastructure  cost. 
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Briefing  Section  II:  DOD’s  Expectations  for 
Advanced  Distributed  Learning  Programs 


A  ^  DOD’s  Expectations  for  ADL 

ADL  Applications  Have  the  Potential  to  Improve 
Readiness 

ADL  has  the  potential  to  improve  readiness  by 

•  Supporting  the  training  transformation  initiative  and 
combatant  commanders. 

•  Enhancing  training  opportunities  for  joint  assignments. 

•  Enhancing  training  opportunities  for  reserve  personnel. 

•  Improving  mission  performance  through  anytime, 
anywhere  and  just-in-time  assignment-oriented  and  job 
performance  enhancement  training. 

•  Improving  manning  by  reducing  personnel’s  non¬ 
availability  and  unit  turbulence,  time  for  in-resident 
training  with  large  return-on-investment  for  temporary 
duty  costs,  while  increasing  retention  and  quality-of-life 
enrichment. 
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Briefing  Section  II:  DOD’s  Expectations  for 
Advanced  Distributed  Learning  Programs 


^_GAO_  DOD’s  Expectations  for  ADL 

ADL  Is  a  Key  to  DOD-wide 
Training  Transformation 

•OSD’s  March  2002  training  transformation  strategy 
emphasizes  ADL  as  a  key  to  meeting  DOD’s  training 
transformation  vision  and  goals  by 

•reengineering  training  and  enhancing  service  members’ 
skills,  and 

•providing  dynamic,  capabilities-based  training  in  support  of 
national  security  requirements  for  service,  joint, 
interagency,  intergovernmental,  and  multinational 
operations. 

•OSD’s  training  transformation  implementation  plan  is  to 
be  completed  by  March  2003. 
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Briefing  Section  III:  Implementation  Status  of 
DOD’s  Advanced  Distributed  Learning 
Programs 


A  Status  of  ADL  Programs 

OSD  Provides  ADL  Program  Policy, 

Oversight  and  Collaboration 

The  Deputy  Under  Secretary  of  Defense  for  Readiness 
(DUSD(R)) 

•Provides  executive  policy  and  programmatic  oversight  and  guidance  for 
the  department’s  ADL  implementation  and 

•Leads  a  collaborative  effort  with  the  services,  Joint  Staff,  combatant 
commands,  other  DOD  components,  the  ADL  co-laboratories,  the  Coast 
Guard  and  Department  of  Labor  to  produce  ADL  policy,  plans,  and 
procedures  for  developing  and  implementing  advanced  distributed 
learning  technologies  across  the  department. 

•  Leads  a  multinational  effort  with  international  military  programs 
such  as  the  NATO  Training  Group. 
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Briefing  Section  III:  Implementation  Status  of 
DOD’s  Advanced  Distributed  Learning 
Programs 


A  ^ Status  of  ADL  Programs 

AccountaOihty  *  Integrily  *  Reliabiltty 

OSD’s  ADL  Vision  and  Strategy  Target 
“Anytime,  Anywhere”  Training 

Vision 

•  To  provide  a  federal  framework  for  using  distributed  learning 
to  provide  the  highest-quality  education  and  training, 
tailored  to  individual  needs,  and  delivered  cost  effectively 
anytime  and  anywhere. 

Strategy 

•  Exploit  existing  network-based  technologies. 

•  Create  interoperable,  reusable  courseware  and  content. 

•  Promote  widespread  collaboration. 

•  Enhance  human  performance. 

•  Promote  common  specifications  and  standards. 

•  Provide  incentives  for  organizational  and  cultural  change. 
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A  Status  of  ADL  Programs 

Accountability  *  integrity  *  Reliability 

OSD  Provides  ADL  Program  Policy, 

Oversight  and  Collaboration 

•  Education  and  Training  Steering  Committee  (ETSC) 

•  Chaired  by  DUSD(R),  provides  executive  policy  and  program 
oversight  and  guidance,  to  help  ensure  creation  of  “anytime, 
anywhere”  learning  environment,  and  membership  is  comprised  of 
one  general  or  flag  officer  from  various  joint  offices,  the  military 
services,  reserve  components,  other  defense  agencies,  and  the  Coast 
Guard. 

•  Total  Force  Advanced  Distributed  Learning  Action  Team 
(TFADLAT) 

•  Working  group  chaired  by  DUSD(R),  Readiness  and  Training,  Plans 
and  Programs,  advises  and  assists  DOD  on  all  aspects  of  ADL,  and 
membership  is  comprised  of  action  officer  personnel  from  various  joint 
offices,  the  military  services,  reserve  components,  other  defense 
agencies,  and  the  Coast  Guard. 

•  Sharable  Content  Object  Reference  Model  (SCORM) 

•  Specifications  adapted  from  multiple  sources  to  provide  a 
comprehensive  suite  of  e-learning  capabilities  that  enable 
interoperability,  accessibility  and  reusability  of  web-based  learning 
content,  initiated  in  coordination  with  the  ADL  co-laboratories  and 
industry  as  the  standard — although  still  evolving-that  DOD  will  use  for 
producing  reusable  learning  content. 
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AccQU'^taPil'ty  *lrit€gri1v  *  Helability 

OSD  Provides  ADL  Program  Policy, 

Oversight  and  Collaboration 

ADL  Co-Labs 

•  Co-sponsored  by  OSD,  the  National  Guard  Bureau,  and  the 
Department  of  Labor,  the  co-labs  were  established  to  provide  an 
open  collaborative  environment  for  sharing  learning  technology 
research,  development,  and  assessments. 

•  ADL  Co-Lab — Located  in  Alexandria,  Va.,  coordinates  DOD,  federal, 
academia,  private-sector,  and  international  research  and  development 
of  ADL  specifications,  guidelines,  policy,  standards,  and  prototypes. 

•  Joint  Co-Lab — Located  in  Orlando,  Fla.  on  site  with  the  military 
services’  training  systems  acquisition  commands  to  support 
cooperative  research;  development;  demonstrations;  and 
assessments  of  ADL  technologies,  tools,  prototypes,  and  guidelines. 

•  Academic  Co-Lab — Located  at  the  University  of  Wisconsin  in 
Madison,  Wis.,  coordinates  and  leverages  research  and  development 
across  academia. 

•  Partnership  Lab  located  in  Telford,  England,  recently  added  to 
collaborative  effort  to  promote  global  e-learning. 
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A  ^  Status  of  ADL  Programs 

Accountability  *  Integrity  *  Reliatallity 

Military  Services’  and  Joint  Staff’s  ADL 
Programs  Generally  Support  DOD’s  Plan 

The  services’  and  Joint  Staff’s  programs  generally  support 
DOD’s  ADL  program  and  training  vision  of  “anytime  and 
anywhere”  for  improved  readiness.  (See  table  3.) 

•  The  services’  and  Joint  Staff’s  programs  in  general  have 

•  very  little  service-specific  or  joint  individual  skill  training 
content; 

•  considerable  technology  access,  although  it  is  far  in 
advance  of  course  content  development  and  conversion 
into  electronic  format;  and 

•  numerous  “contractor-off-the-shelf”  content  and 
correspondence  courses  available. 
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Table  3:  Comparison  of  DOD,  the  Military  Service  and  Joint  Staff  Programs 


Vision 

Strategy 

Implementation 

Success 

Challenges 

Funding  (Millions)  | 

FY  99-02 

POM  FY  03-07 

OSD 

Framework  for 

Network  based  technology 

ETSC 

SCORM 

Funding 

$48 

$70 

Anytime 

Collaboration 

TFADLAT 

ADL  Co-labs 

Metrics,  Content 

Anywhere 

Common  standards 

ADL  Co-labs 

Plugfests 

Culture 

Reusable  courseware 

SCORM 

Army 

Individually  Tailored 

Internet,  CD,  VTC 

Fielding  Digital  Training 

Battle  Staff  NCO 

Funding 

$285 

$906 

Training 

Online  Learning  Management 

Facilities 

Course 

Culture-school  houses 

Anytime 

System 

Redesigning  Courseware 

UFI-60  Repairer 

Technology-content 

Anyw  here 

Fielding  Deployable  Systems 

for  active,  reserve,  NGB 

Transition  Course 

Navy 

Quality  Training  to 

Enterprise  Strategic  System 

.mil  and  .com  sites 

Interoperable  C4I 

Funding-Not  sufficient 

$27 

$462 

Right  People 

System  Architecture 

Deploy  Learning  Mgmt. 

Content 

to  run  content 

Right  Tme 

Evaluating  Prototypes 

System 

Medical  Courseware 

Technology-accelerak 

Right  Race 

Course  conversions 

Marine  Corps 

World  Qass  Training 

Automated  electronic  dassrooms 

Developing  Deployable 

Terrorism  Awareness 

Funding-  mgmt. 

$58 

$93 

via  MC  learning  network 

Video  TeleTraining 

Learning  Resource  Centers 

Basic  Electronics 

Technology-access 

Where  and  w  hen  needed 

Learning  Resource  Centers 

Convert  courseware  for  PME 

Improve  Efficiency 

Culture-schoolhouse 

and  MOS  courses 

&  Effectiveness 

Air  Force 

Leverage  Technology  to 

strategy  and  Implementation 

AFIADL  as  Executive  agent 

Funding  not  sufficien 

$13 

$29 

provide  the  right  training 

plans  being  developed 

Print  based,  interactive  TV 

Firefighter  course 

to  field  program 

and  education  anytime, 

Emphasize  centralized  vision 

LMS  operational  for 

Culture 

anywhere 

direction  and  standardization 

internet-based  ADL 

Technology 

Joint  Staff 

Increase  Joint  Readiness 

Develop  and  publish  Joint  ADL 

Devetoping  AIX  courses 

Joint  ADL  portal 

Culture-Joint  vs.  Servia 

$0 

$0 

Access  to  high  quality  joint 

policy 

Develop  linkages  from 

prototype 

Regulatory-No  directives 

opportunities 

Develop  content  architecture 

combatant  commanders  to 

Reserve  officers 

Funding-Inadequate  for 

and  integration  of  systems 

joint  learning  requirements 

complete  JPME  II 

implementation 

Source:  DOD. 

Note:  GAO  analysis  of  OSD,  Joint  Staff,  and  military  service  data. 

The  Joint  Staff  reportedly  added  $650,000  per  year  to  its  fiscal  years  2003  through 
2007  POM  after  we  completed  our  audit  work. 
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OSD  and  the  Services  Have  Achieved 
ADL  Program  Successes  - 1 

•OSD 

According  to  ADL  Program  officials: 

•SCORM  standard  for  producing  interoperable  courseware  is  working  but 
still  evolving,  and  all  the  services  are  willing  to  develop  content  that  is 
SCORM  compliant. 

•ADL  co-labs  have  generated  successful  prototypes  for  OSD,  the  services. 

Joint  Staff,  and  co-sponsors. 

•  The  Joint  Staff’s  Joint  Collaborative  Learning  Environment  (JCLE)  prototype 
established  the  initial  joint  personnel  tracking  and  portal  prototype  capability. 

•OSD  and  co-lab  sponsored  “Plugfests”  offer  a  venue  where  all  ADL 
partners  could  test  and  synchronize  content  authoring  tools,  learning 
management  systems,  and  web-based  course  content  to  SCORM. 

•Joint  Staff 

According  to  Joint  Staff  officials: 

•ADL  portal  prototype  established. 

•Joint  Operations  course  developed,  delivered  and  evaluated. 

•Through  U.S.  Joint  Forces  Command  developed  web-based  joint  training 
to  prepare  for  Joint  Task  Force  Headquarters  Staff  and  Combatant 
Commander  Battle  Staff  exercises. 
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A  ^  Status  of  ADL  Programs 

Accountability  *  Integrity  *  Reliability 

OSD  and  the  Services  Have  Achieved 
ADL  Program  Successes  -  2 

•Services 

•Army 

According  to  Army  ADL  officials,  Battle  Staff  Noncommissioned 
Officer  (NCO)  course,  a  two-phased  training  approach- 
asynchronous  ADL  instruction  at  soldiers’  location  and 
residence;  training  or  ADL  via  Video  Tele  Training  (VTT)  at 
remote  locations — resulted  in  a  $2.9  million  annual  cost 
avoidance  and  soldiers  perform  equal  to  resident  course. 

•Navy 

According  to  Navy  ADL  officials.  E-learning  courses,  now 
available  on  .mil  and  .com,  promote  interoperability,  efficient 
distribution,  and  reduced  training  time. 
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^_GAO_  Status  of  ADL  Programs 

Accountability  *  Intg^rily  *  HeliabMity 

OSD  and  the  Services  Have  Achieved 
ADL  Program  Successes  -  3 

•Services 

•Marine  Corps 

According  to  Marine  Corps  ADL  officials,  DL  application  in 
Terrorism  Awareness  course  reduced  training  time  from  1 1  to  6 
hours  and  increased  average  exam  scores  from  85  to  92 
percent. 

•Air  Force 

According  to  Air  Force  ADL  officials,  a  CD-ROM  based 
hazardous  material  incident  response  training  course  for  DOD 
firefighters  and  law  enforcement  personnel,  resulted  in  a 
significant  increase  of  certified  responders,  and  a  projected 
$16.6  million  cost  avoidance. 
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^_GA^  Status  of  ADL  Programs 

Accountability  *  Integrity  *  ReliaDility 

OSD  and  the  Services  Have  Achieved 
ADL  Program  Successes  -  4 

•Other  DOD 

•Defense  Acquisition  University  (DAU) 

•ADL  program  is  cited  as  a  success  and  an  example  of  “best 
practices.”  Awarded  the  U.S.  Distance  Learning  Association  Award 
for  Excellence  in  Government  (2001  and  2002)  for  quality  of  its  19 
on-line  offerings. 

•  According  to  DAU  ADL  officials: 

•  On-line  program  management  curriculum  reduced  annual  student 
training  weeks  from  36,120  to  10,000 — a  real  savings  of  300  annual 
work  years  or  $17.4  million. 

•  Since  1998: 

•graduates  attending  on-line  training  courses  increased  38  percent; 

•on-line  instructional  time  increased  from  15,750  to  1 .4  million  hours  resulting  in 
a  48%  increase  in  student  throughput,  40  percent  decrease  in  student  travel 
costs,  and  24  percent  decrease  in  faculty  and  staff. 
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A  ^ ADL  Program  Challenges 

Accountabilfty  *  Integrity  *  Relwftility 

Challenges  Affect  OSD’s,  the  Services’, 
and  Joint  Staff’s  ADL  Execution 

Various  cultural,  technological,  policy,  and 
financial  challenges  affect  OSD’s,  the  services’, 
and  the  Joint  Staff’s  ability  to  execute  programs 
that  fully  achieve  their  ADL  vision  and  training 
transformation  strategies. 
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^_GA^  ADL  Program  Challenges 

Accountability  *  Integrity  *  Reliability 

Cultural  Challenges 

According  to  DOD  ADL  officials,  top  down  commitment  varies, 
and  focus  of  traditional  schoolhouse  learning  culture  is  difficult 
to  overcome. 

•Top  military  and  civilian  management  commitment  is  not  always 
apparent  in  all  the  services;  unit  commanders  and  civilian  managers 
resist  allowing  daily  ADL  training  time;  and  reserve  components  have 
not  always  provided  ADL  training  time  during  normal  training  periods 
or  additional  periods  in  a  paid  status. 

•School houses  are  most  resistant  to  change  because  their  current 
infrastructure — funding,  faculty,  and  facilities — is  closely  tied  to 
student  throughput. 

•Lack  of  widespread  computer  literacy  among  older  military  and 
civilian  decision  makers  is  a  barrier  to  implementing  ADL. 


Page  29 


GAO-03-393  DOD's  ADL  Programs 


Briefing  Section  IV:  Major  Challenges 
Affecting  DOD’s  Advanced  Distributed 
Learning  Programs 


A  GAO  aDL  Program  Challenges 

Accpurvtabilfty  *  Integrity  *  Reliability 

Technological  Challenges 

Existing  course  content  and  technology  does  not  always 
support  “anytime,  anywhere”  delivery. 

•  Bandwidth  is  often  insufficient  to  support  desired  use  of  multi- 
media  interactive  courseware,  especially  for  deployed  soldiers. 

•  Emphasis  on  securing  networks  may  impede  learner’s  ability  to 
access  education  and  training  anytime,  anywhere. 

•  ADL  standards  and  specifications  are  still  evolving  and  it  is 
difficult  for  programs  to  keep  pace. 

•  Good  military  occupational  skill  related  learning  content  is  crucial 
to  program  success  but  currently  there  is  little  available  because 
content  is  more  complicated  to  develop  than  expected. 
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Accountability  *  Irit€-gri1\  *  Reliability 

Policy  Challenges 

DOD  is  in  the  eariy  stages  of  formuiating  poiicy  that 
specificaiiy  addresses  the  use  of  ADL.  DOD  ADL 
program  officiais  stated: 

•Some  of  DOD’s  training  policies  are  obsolete  and  consequently 
some  of  the  military  services’  training  regulations  do  not  reflect 
the  availability  or  use  of  new  ADL  technologies. 

•Without  specific  OSD  policy  or  guidance,  the  services  do  not 
have  a  requirement  to  change  or  update  their  regulations  to 
reflect  the  availability  of  or  use  of  new  ADL  technologies. 

•Changes  occur  at  a  very  slow  pace. 
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Accountability  *  Integrity  *  Reliability 

Financial  Challenges  - 1 


OSD^  the  Joint  Staff  and  the  services  allocated  more  than  $431 
million-only  about  1.3%  of  the  total  training  budget-to  irnplement 
ADL  programs  during  fiscal  years  1999  through 2002.  (See  table 
4.) 

•  Funds  allocated  did  not  always  meet  program  needs  (content 
development  and  delivery  infrastructure),  although  allocations  at  times 
equaled  or  exceeded  funding  requests.  (See  table  5.) 

•  Program  and  consequently  funding  requirements  are  difficult  to  determine  for 
a  new  advanced  technology  program  where  standards  are  evolving  and  the 
technology  changes  rapidly. 

•  Prior  to  fiscal  year  2003  the  Joint  Staff  and  the  services  had  no  single 
accounting  code  to  account  for  the  total  amount  of  funding  spent  to  develop 
and  implement  ADL  projects  and  programs. 

•  Allocated  funding  mix  did  not  always  work  well  for  a  new  program. 

•  O&M  funding — available  for  obligation  for  only  1  year — is  difficult  to  manage, 
especially  for  courseware  development  which  often  exceeds  18  months  and 
costs  more  than  anticipated. 

•  Procurement  funding-allocations  did  not  always  meet  needs  because  of 
significant  up-front  investment  for  facilities  and  equipment  required  to  realize 
long-term  return  on  investment. 
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Table  4:  Training  and  ADL  Allocations  By  Component  for  Fiscal  Years  1999  through  2002  (in  Millions) 


FY  99 

FY  00 

FY  01 

FY  02 

FY  99-02 

Component 

Allocation 

Allocation 

Allocation 

Allocation 

Total 

Total  DoD 

Total  Training 

ADL 

ADL  Percent  of  Total  Training 

$7,532 

$88 

1.17% 

$7,973 

$94 

1.18% 

$8,868 

$116 

1.31% 

$9,361 

$133 

1 .42% 

$33,734 

$431 

1 .28% 

Army 

Total  Training 

ADL 

ADL  Percent  of  Total  Training 

$3,239 

$60 

1.85% 

$3,394 

$58 

1.71% 

$3,762 

$75 

1 .99% 

$4,047 

$92 

2.27% 

$14,442 

$285 

1 .97% 

Navy 

Total  Training 

ADL 

ADL  Percent  of  Total  Training 

$1,789 

$6 

0.34% 

$1,862 

$4 

0.21% 

$2,064 

$7 

0.34% 

$2,135 

$10 

0.47% 

$7,850 

$27 

0.34% 

Marine  Corps 

Total  Training 

ADL 

ADL  Percent  of  Total  Training 

$433 

$10 

2.31% 

$450 

$14 

3.11% 

$478 

$16 

3.35% 

$478 

$18 

3.77% 

$1 ,839 
$58 

3.15% 

Air  Force 

Total  Training 

ADL 

ADL  Percent  of  Total  Training 

$1,870 

$2 

0.11% 

$2,033 

$4 

0.20% 

$2,304 

$4 

0.17% 

$2,439 

$3 

0.12% 

$8,646 

$13 

0,15% 

OSD  and  Joint  Staff 

Total  Training 

ADL 

ADL  Percent  of  Total  Training 

$201 

$10 

4.98% 

$234 

$14 

5.98% 

$260 

$14 

5.38% 

$262 

$10 

3.82% 

$957 

$48 

5.02% 

Source:  DOD. 


Notes:  GAO  analysis  of  OSD,  Joint  Staff  and  the  military  service  budgetary  data. 

Total  Training  amount  includes  all  component  O&M  training  funding  allocated  as  reported 
in  the  DOD  budget  for  Budget  Activity  3  (BAS)  for  the  indicated  fiscal  years.  Budget  Activity 
3  funds  all  training  and  recruiting  programs. 

Reserve  component  funding  is  included  within  the  active  duty  component  totals. 
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Table  5:  ADL  Funding  History  by  DoD  Components 


Component 

Category  of 
Funding 

FY  99 

FY  2000 

FY  2001 

FY  2002 

Total  FY  99-02  | 

Budget 

Request 

Alloca¬ 

tion 

%  Differ¬ 
ence 

Budget 

Request 

Alloca¬ 

tion 

%  Differ¬ 
ence 

Budget 

Request 

Alloca¬ 

tion 

%  Differ¬ 
ence 

Budget 

Request 

Alloca¬ 

tion 

%  Differ¬ 
ence 

Budget 

Request 

Alloca¬ 

tion 

%  Differ¬ 
ence 

DoD 

Grand  Total 

$79.1 

112.01% 

$78.4 

$93.3 

119.01% 

$92.0 

$115.0 

b 

o 

$138.0 

$133.1 

96.45% 

$387.5 

$430.0 

110.97% 

Army 

O&M 

$41.8 

$42.2 

100.96% 

$51.4 

$46.2 

89.88% 

$46.2 

$44.3 

95.89% 

$92.2 

$64.7 

70.17% 

$231.6 

$197.4 

85.23% 

Procurement 

$30.4 

$17.7 

58.22% 

$17.8 

$11.6 

65.17% 

$28.0 

$24.6 

87.86% 

$28.1 

$25.1 

89.32% 

$104.3 

$79.0 

75.74% 

R,D,T&E 

$0.0 

$0.0 

$0.0 

$0.0 

$4.9 

$5.8 

118.37% 

$2.2 

$2.1 

95.45% 

$7.1 

$7.9 

111.27% 

Subtotal 

$72.2 

$59.9 

82.96% 

$69.2 

$57.8 

83.53% 

$79.1 

$74.7 

94.44% 

$122.5 

$91.9 

75.02% 

$343.0 

$284.3 

82.89% 

Navy 

m 

O&M 

$6.9 

$6.3 

91.30% 

$4.1 

44.57% 

$12.9 

$7.3 

56.59% 

$13.0 

$10.3 

79.23% 

$42.0 

$28.0 

66.67% 

Subtotal 

$6.9 

$6.3 

91.30% 

$4.1 

44.57% 

$12.9 

$7.3 

56.59% 

$13.0 

$10.3 

79.23% 

$42.0 

$28.0 

66.67% 

Marine  Corps 

O&M 

$0.0 

$10.3 

0.00% 

$0.0 

$7.9 

0.00% 

$0.0 

$9.1 

0.00% 

$0.0 

$11.9 

0.00% 

$0.0 

$39.2 

0.00% 

Procurement 

$0.0 

$0.0 

$0.0 

$6.5 

0.00% 

$0.0 

$6.9 

0.00% 

$0.0 

$6.5 

0.00% 

$0.0 

$19.9 

0.00% 

Subtotal 

$0.0 

$10.3 

0.00% 

$0.0 

$14.4 

0.00% 

$0.0 

$16.0 

0.00% 

$0.0 

$18.4 

0.00% 

$0.0 

$59.1 

0.00% 

Air  Force 

O&M 

$0.0 

$2.1 

0.00% 

$0.0 

$3.4 

0.00% 

$0.0 

$3.5 

0.00% 

$2.5 

$2.5 

100.00% 

$11.5 

460.00% 

Procurement 

$0.0 

$0.0 

$0.0 

$0.1 

0.00% 

$0.0 

$0.0 

$0.0 

$0.0 

$0.1 

0.00% 

Subtotal 

$0.0 

$2.1 

0.00% 

$0.0 

$3.5 

0.00% 

$0.0 

$3.5 

0.00% 

$2.5 

$2.5 

100.00% 

$11.6 

464.00% 

OSD  &  Joint  Staff 

R,DJ&E 

$0.0 

$10.0 

0.00% 

$0.0 

$13.5 

0.00% 

$0.0 

$13.5 

0.00% 

$0.0 

$10.0 

0.00% 

$0.0 

$47.0 

0.00% 

Subtotal 

$0.0 

$10.0 

0.00% 

$0.0 

$13.5 

0.00% 

$0.0 

$13.5 

0.00% 

$0.0 

$10.0 

0.00% 

$0.0 

$47.0 

0.00% 

Source:  DOD. 


Notes:  GAO’s  analysis  of  OSD,  Joint  Staff,  and  military  service  budgetary  data. 

Reserve  component  funding  amounts  were  included  with  the  active  component 
funding  data. 
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#  G  AO  ADL  Program  Challenges 

Accountabil'ty  *  Intfigniy  *  H&l'3Gintv 

Financial  Challenges  -  2 


•  Future  funding  levels  may  not  meet  future  program 
requirements.  (See  figure  1.) 

•  OSD,  the  Joint  Staff  and  the  services  project  they  will  need 
more  than  $2.2  billion  to  implement  ADL  programs  for  fiscal 
years  2003  through  2007,  but  have  planned  to  budget  for  only 
about  $1.6  billion-$626  million  less  than  projected 
requirements. 

•  The  Joint  Staffs’  program  is  dependent  upon  the  combatant 
commanders  for  funding. 

•  Anticipated  return  on  investment  savings  have  been  taken 
from  the  programs  before  they  were  realized. 

•  Services  are  still  trying  to  determine  how  to  budget  for  long¬ 
term  ADL  use  (currently  budgets  are  still  based  on  student 
throughput). 
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ADL  Program  Challenges 

Total  DOD,  Military  Services,  and  Joint  Staff  Requirements  Compared  to  Programmed  Requirements  for  FY  03-07 


2500  Dollars  in  millions 


200 


2000 


1500 


1000 


500 


0 


2185.53 


150 


132.83 


DOD  Army  Navy 


Total  Requirements 

$2185.53 

$1298.50 

$590.90 

POM 

$1559.68 

$905.62 

$462.10 

Total  Difference 

-$625.85 

-$392.88 

-$128.80 

Marine  Corps  Air  Force  OSD  &  Joint  Staff 


$93.30 

$132.83 

$70.00 

$93.30 

$28.66 

$70.00 

-$0.00 

-$104.17 

-$0.00 

I  I  Infrastructure  =  all  hardware  costs  associated  with  ADL  programs  e.g.  (automated  electronic  classrooms, 
computers,  networking  equipment,  servers.  Learning  Management  Systems,  etc  ). 


Content  =  all  costs  required  to  develop  the  courseware  for  the  ADL  program. 

I  I  POM  =  Program  objective  memorandum-that  portion  of  total  ADL  projected  requirements  for  which 
DOD,  the  services  and  joint  staff  plan  to  request  budgetary  resources. 


Source:  DOD. 


Notes:  GAO  analysis  of  OSD,  military  service,  and  Joint  Staff  budgetary  data. 

The  DOD  bars  reflect  the  total  requirements  of  the  services,  OSD  and  Joint  Staff. 

Total  requirements  include  both  infrastructure  and  content  requirements. 

Joint  Staff  requirements  are  included  in  the  “OSD  and  Joint  Staff”  total  because  Joint  Staff 
receives,  funding  for  ADL  projects  from  the  funds  allocated  to  OSD  for  ADL  projects.  The 
Joint  Staff,  along  with  the  military  services,  competes  for  funds  allocated  to  OSD  for  ADL 
prototypes  projects. 
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^  G  AO  Conclusions 

Accountability  *  Int^gnly  *  Reliability 


•The  need  to  transform  training  within  DOD  has  been  demonstrated  over 
time  by  its  difficulties  in  sustaining  unit  and  individual  readiness.  The 
scope  of  OSD’s,  the  services’  and  Joint  Staff’s  ADL  programs 
demonstrates  the  extent  to  which  they  see  ADL  as  a  means  to  save 
money. 

•ADL  appears  to  offer  opportunities  for  substantial  learning  and 
performance  enhancements,  anytime  and  anywhere.  Enhanced 
learning,  performance  and  readiness  opportunities  may  outweigh  simple 
return-on-investment  calculations.  Developing  good,  relevant  and 
military  occupational  skills-related  content  that  improves  both  learning 
and  readiness  is  crucial  to  ADL  program  implementation  success. 

•Existing  cultural,  technological,  policy,  and  financial  challenges,  create 
implementation  barriers  for  DOD  leadership  to  address  before  it  can 
fully  reap  what  appear  to  be  achievable  benefits  of  enhanced  learning 
and  performance  and  improved  readiness. 
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A  GAO  Army’s  ADL  Plan 

The  Army  Distributed  Learning 
Program  (TADLP) 

Vision 

“Deliver  high  quality,  individually  tailored,  and  cost  effective  training  and 
education  to  soldiers  and  leaders  anywhere,  anytime  through  an 
integrated  network  of  technological  capabilities  and  processes.” 

Strategy 

•  Initiated  in  April  1996,  acquisition  program  coordinated  with  U.S.  Army 
Training  and  Doctrine  Command  (TRADOC). 

•  Provide  improved  access,  reach  back  capability  and  lifelong  learning. 

•  Complete  implementation  by  FY  2010. 

•  Fielding  digital  and  deployable  digital  training  facilities  throughout  the 
world  to  ultimately  provide  95%  force  coverage. 

•  Redesigning  standardized  Army  courses  for  various  DL  delivery  via 
the  Internet,  CD-ROM,  and  VTC  when  needed. 

•  Implementing  an  on-line  learning  management  system  to  track 
students  through  their  courses. 
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GAO 

Accountability  *  Integrity  *  R&lia&llity 


Implementation 


Army’s  ADL  Plan 


TADLP  covers  Active,  National  Guard,  and  Reserve 
components 

•  Digital  Training  Facilities  (DTFs) 

•  Goal  -  850  DTFs  to  cover  95%  of  the  force 

•  Completed  -  522;  207  Army  DTFs  and  315  DTTP  DTFs 

•  Courseware  Development 

•  Goal  -  525  courses  at  47  per  year  by  FY  2010,  for  Active 
Component,  Reserve  Component  and  civilians 

•  Completed  -  63;  43  courses  developed  through  TRADOC,  20 
courses  through  National  Guard  completed. 
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A  GAO  Army’s  ADL  Plan 

Army  National  Guard  Bureau  (NGB) 

Distributive  Training  Technology  Project  (DTTP) 

Vision 

•  “Anywhere,  anytime”  distance  learning  for  improved  readiness. 

•  Command  and  control. 

•  Shared  use  with  local  communities. 

Strategy 

•  Initiated  in  1995,  DTTP  network  established  with  Congressional 
plus-up  funding. 

•  Complete  implementation  by  FY  2010,  coordinating  with 
TRADOC  in  fielding  facilities  and  designing  courseware. 
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GAO 

Accountability  *  Integrity  *  Reliability 


Army’s  ADL  Plan 


other  Army  Distance  Learning  (DL) 
Programs 


•Army  University  Access  On-line  (eArmyU) 

•Provided  computers  and  Internet  access  to  registered  soldiers 
to  take  on-line  courses  at  no-cost. 

•Army  Correspondence  Course  Program 

•Self  development  training  -  300  courses  and  1 ,800 
subcourses  available. 

•Army  War  College  -  Department  of  Distance  Education 

•Two  year  program  of  ten  correspondence  courses  and  two 
resident  courses. 

•Computer  Based  Training  and  “Smartforce”  Courses 

•Information  technology,  business,  and  interpersonal  skills 
training  -  1,500  courses. 
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Program  Successes 

•  Battle  Staff  NCO  Course  -  Army 

•  Combines  asynchronous  instruction  at  soldiers’  location  with 
resident  schoolhouse  training  or  DL  via  VTT  at  remote 
locations. 

•  Based  on  an  Army  Research  Institute  study,  soldiers’ 
performance  was  equal  to  resident  course  based  on 
performance  assessment. 

•  Cost  $759  thousand  to  develop;  annual  cost  avoidance  of  $2.9 
million. 

•  UH-60  Helicopter  Repairer  Transition  -  Army  NGB 

•  7-week  resident  course  converted  to  1 -weekend  training  period 
for  5  months  and  a  2-week  training  period  using  VTC. 

•  Additional  52  new  mechanics  qualified  over  concurrent  resident 
training;  improved  state’s  combat  readiness  posture. 
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4  GAO  Army’s  ADL  Plan 

Challenges 

Cultural 

•  Acceptance  tied  to  age  of  user  and  computer  literacy. 

•  School  houses  most  reluctant  to  change. 

•  Top-down  commitment. 

Technological 

•  Courseware  developed  and  fielded  slower  than  desired. 

•  Good  military  occupational  skill  related  content  is  important  to 
program  success  but  there  is  little  available  because  it  is  more 
complicated  to  develop  than  expected. 

•  Contracting  difficulties. 

Regulations 

•  Do  not  incorporate  ADL  and  have  changed  at  a  very  slow  pace. 

Financial 

•  Early  harvesting  of  anticipated  savings  before  realized. 

•  Lack  of  current  funding  to  meet  requirements. 
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A  GAO  Navy’s  ADL  Plan 

Vision 

“To  deliver  quality  education  and  training— to  the  right  people, 
at  the  right  time,  and  at  the  right  place— as  part  of  a  career 
long  training  continuum  supporting  Navy  operational  readiness 
and  personal  excellence.” 

strategy 

•  Developed  in  December  1998,  serves  all  active  duty,  reserve, 
and  Department  of  the  Navy  civilian  employees. 

•  Five  Phase  implementation  plan. 

•  I  Develop  Enterprise  Strategic  Plan. 

•  II  Refine  Enterprise  Strategic  Plan. 

•  III  Design  and  develop  system  architecture. 

•  IV  Evaluate  prototype. 

•  V  Incorporate  enhancements  prior  to  fielding. 
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A  Navy’s  ADL  Plan 

Implementation 

•  Program  fielded  in  May  2001  with  .com  and  .mil  websites;  classified 
site  expected  in  early  fiscal  year  2003. 

•  Working  to  expand  course  content,  deploy  the  learning  management 
system  (LMS)  regionally,  and  provide  similar  accessibility  to  afloat 
community. 

•  Course  conversion  concentrated  on  Navy-specific  courses. 

Courses 

•  Currently  offered: 

•  50  Navy-specific  courses, 

•  800  NetG  technical  courses, 

•  580  Skillsoft  business  and  management  courses. 

•  Course  conversion  and  LMS  implementation  have  been  slow,  but 
progressing  in  a  phased  approach. 

•  Chief  Naval  Education  and  Training  (CNET)  projected  25  new  Navy 
courses  per  year.  50  have  been  completed  since  May  2001 . 
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^  G  AO  Navy’s  ADL  Plan 

Program  Successes 

Navy  E-Learning  Courses 

•  C4I  Content  for  Cryptologic  Technicians. 

•  Using  the  same  schoolhouse  curricula,  produced  to 
shorten  the  time  taken  for  facilitator  stand-up  training. 

•  Interoperable  between  a  number  of  ratings  and  skill 
sets. 

•  Differentiation  between  Chemical,  Biological,  and 
Radiological  Casualties  (Navy  Medical). 

•  Presented  through  Navy  E-Learning  because  CD 
distribution  and/or  resident  training  was  inefficient. 

•  High  importance  to  the  Naval  Surgeon  General. 
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A  Navy’s  ADL  Plan 

Challenges 

Technological 

•  Course  conversion  and  LMS  implementation  have  kept  pace  with 
execution  year  funding-need  to  accelerate. 

•  Small  ship  communications  infrastructure  upgrades,  to  permit 
NIPRNET  and  SIPRNET  access. 

•  Bandwidth  limitations  for  deployed  units. 

Financial 

•  Funding  gap  for  course  development  funding. 

•  Need  additional  funding  for  classroom  hardware  to  run  content — Navy 
and  Marine  Corps  Intranet  (NMCI)  hardware  is  inadequate  to  execute 
all  web  based  training  course  content. 

•  Anticipated  return-on-investment  savings  are  notably  less  than 
anticipated  life  cycle  costs. 

•  Significant  up-front  costs  are  required  to  achieve  life-cycle  costs. 
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^  G  AO  Marine  Corps’  ADL  Plan 

Vision 

“Deliver  world-class  training  and  education  ...  enabling 
Marines  (active,  reserve,  and  civilians)  to  learn  via  the 
appropriate  media,  when  and  where  learning  is  most 
needed.  The  learning  experience  will  be  part  of  a  Marine’s 
career-long  learning  continuum  that  supports  the  operational 
readiness  of  the  Total  Force...” 

strategy 

Initiated  in  fiscal  year  1997,  establish  a  network  of 
Automated  Electronic  Classrooms  (AEG),  Video 
TeleTraining,  Learning  Resources  Centers,  and  Deployable 
Learning  Resources  Centers  that  utilize  ADL  technology  to 
provide  the  total  force  with  training  wherever  and  whenever 
they  require  it. 
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Marine  Corps’  ADL  Plan 


Implementation 

•  Program  delivers  anytime,  anywhere  training  and  education  with 
emphasis  on  right  time,  right  place  with  appropriate  media. 

•  .com  connection  not  yet  implemented  to  facilitate  at  home  training- 
but  scheduled  for  implementation  by  December  2003. 

•  Developing  Deployable  Learning  Resource  Centers  (DLRC)  for 
deployed  Marines,  compatible  with  Navy  Shipboard  Network. 

•  Developing  web  compatible  course  content  in  absence  of  an 
evolving  SCORM  standard-will  have  to  retool  some  courses  once 
SCORM  is  more  robust  to  ensure  durability. 

Courses 

•  Focused  on  Professional  Military  Education  (PME)  and  military 
occupational  specialty  (MOS)  courses  or  traditional  classroom 
training. 

•  800  courses  currently  available; 

•  25  Marine  Corps-specific  courses  are  available  now, 

•  Remaining  courses  are  Commercial  IT  (Smartforce)  and  Skillsoft 
business/management  courses. 

•  32  Marine  Corps-specific  courses  will  be  available  soon. 

•  Minimal  or  no  impact  on  entry-level  and  unit-level  training. 
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^  G  AO  Marine  Corps’  ADL  Plan 

Program  Successes 

•  Terrorism  Awareness  Course  -  updated  from 
paper-based  to  interactive  muitimedia  instruction. 

•  Improved  Efficiency  -  Time  to  complete  course  reduced 
from  1 1  hours  to  6  hours. 

•  Improved  Effectiveness  -  Average  exam  score 
increased  from  85%  to  92%. 

•  Basic  Electronic  Theory  and  Labs  -  updated  with 
multimedia  software  and  appended  training 
devices 

•  Improved  Efficiency  -  Course  length  reduced  from  55 
days  to  40  days. 

•  Improved  Effectiveness  -  Failure/recycle  rates  reduced 
from  46%  to  6%. 
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Marine  Corps’  ADL  Plan 


Challenges 

Cultural 

•  Resistance  to  change  compounded  by  lack  of  personal  experience 
with  DL  technology. 

•  Success  of  DL  tied  to  developing  incentives  for  completing  DL 
courses  to  ensure  utilization. 

•  School  house  culture  most  reluctant  to  change. 

Technological 

•  NMCI  computer  purchase  moratorium  delaying  Automated  Electronic 
Classrooms  deployment. 

•  Strict  Marine  Corps  firewall  policies  may  limit  access. 

•  Course  conversion  process  can  take  longer  than  expected. 

Financial 

•  O&M  funding  for  ADL  is  difficult  to  manage  because  it  is  available  for 
obligation  only  1  year-lead  time  for  course  conversion  can  exceed  18 
months. 

•  Program  requires  significant  up-front  procurement  costs  to  realize 
long-term  return  on  investment. 
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A  Air  Force’s  ADL  Plan 

Accountability  *  Irtegnts*  *  Hehability 

Vision 

•  Leverage  technology  to  provide  the  right  training  and 
education  anytime,  anywhere. 

•  Envisions  enterprise-wide  ADL  applications  that  improve 
readiness,  provide  efficient  delivery,  and  support  the  Air  and 
Space  Expeditionary  Force  and  force  development  construct. 

strategy 

•  Program  intended  to  serve  all  active  duty,  air  reserve 
component,  and  Air  Force  civilian  employees  and  support  Air 
Force  goals  for  its  force  management  and  force  development 
construct. 

•  Air  Force  strategic  plan  and  proposed  implementation  plan 
will  result  in  centralized  vision  and  direction,  and 
standardization  of  ADL  roles  and  responsibilities. 
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A  Air  Force’s  ADL  Plan 

Accou'it3t>il'ty  *  Irtegnly  *  Helability 

Implementation 

•  Air  Force  Institute  for  Advanced  Distributed  Learning  (AFIADL), 
initiated  in  February  2000,  is  focal  point  for  ADL  activities,  including 
print-based  and  interactive  television  activities;  print-based  distance 
learning  activities  performed  since  1950  and  interactive  television 
activities  since  1991. 

•  There  is  no  centrally  funded  Air  Force  ADL  system,  learning 
management  system  is  operational  for  Internet-based  ADL,  and 
enterprise-wide  implementation  depends  on  course  content 
standards  and  technology  infrastructure  development. 

Courses 

•  Air  Education  and  Training  Command  (AETC)-105  courses 

•  Extension  Course  Program  (ECP)-439  courses. 

•  Air  Technology  Network  (ITV)-3,500  hours. 

•  Computer  Based  Training  (CBT)-over  1 ,400  courses. 

•  In  fiscal  year  1998,  AETC  contracted  evaluation  of  its  technical 
training  and  Air  University  Courses  for  DL  conversion:  Anajyzed 
1,154  courses  and  identified  128  for  conversion;  approved  /4 
courses  for  conversion,  or  technology  insertion  in  fiscal  years  2000 
through  2007;  converted  13  technical  training  courses  and  20  Air 
University  courses. 
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Air  Force’s  ADL  Plan 

Accountebilfty  *  Integrity  *  Relwbllity 

Program  Successes 


Air  Force  Civil  Engineer  Support  Agency 

•  Needed  to  train  and  certify  a  large  number  of  emergency 
responders  in  hazardous  material  training.  Training 
capability  was  1 ,500  students  per  year,  available  only  at 
the  Air  Force  Firefighting  School  in  the  basic  firefighter 
course. 

•  Developed  series  of  CD-ROMs  at  cost  of  $1  million. 
Projected  cost  avoidance  of  $16.6  million  (compared  to 
pursuing  continued  contract  training). 

•  Trained  DOD  firefighters,  law  enforcement  and  security 
police,  with  about  60,000  certificates  awarded  since 
August  1994. 

•  Utilized  a  cooperative  agreement  that  allows  the  product  to 
be  packaged  and  sold  to  the  commercial  sector, 
generating  8-9  %  profits  for  the  Air  Force. 
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Air  Force’s  ADL  Plan 

Accou'>taDility  *  Ir-tegri1y  •  Reliability 

Challenges 

Cultural 

•  Overcoming  existing  learning  culture  is  difficult. 

Technological 

•  Current  technical  infrastructure  and  bandwidth  need  to  be 
evaluated  to  determine  future  ADL  needs. 

•  Many  existing  ADL  applications  are  “just-in-time”  and  need 
to  be  standardized. 

•  Course  conversion  process  slow. 

Financial 

•  Funding  has  not  always  met  program  vision. 
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Briefing  Section  X:  Joint  Staffs  Advanced 
Distributed  Learning  Programs 


A  gap  Joint  Staff’s  ADL  Plan 

Accountability  *  Ir.tBgnty  *  Hel  ability 

Vision 

To  increase  joint  readiness  by  providing  access  to  the 
highest  quality  joint  training  opportunities  through  the 
exploitation  of  advanced  distributed  learning,  learning 
enablers  and  resident  instruction. 

strategy 

•Establish  an  information  system  that  effectively  supports 
Distributed  Learning  anytime,  anywhere  through  the 
appropriate  exploitation  of  information  technology  across  all 
joint  areas  (e.g.  training,  personnel,  doctrine,  standard 
operating  procedures,  campaign  planning,  operational  art, 
lessons  learned,  etc.). 

•Effort  began  in  fiscal  year  1999;  prototype  site  initiated  in 
fiscal  year  2001. 
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Distributed  Learning  Programs 


i,  GAO 

Joint  Staffs  ADL  Plan 

AccDu^tabilfty  *  Integrity  *  Reliability 

Implementation 


•Develop  and  Publish  Joint  ADL  policy. 

•  Advance  the  Joint  Advanced  Distributed 
Learning  Portal  and  integrate  DOD,  Joint,  and 
Service  ADL  systems. 

•  Coordinate  transformation  of  joint  courseware; 
currently  there  are  no  ADL  courses. 

•  Develop  Joint  core  curriculum. 
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Briefing  Section  X:  Joint  Staff’s  Advanced 
Distributed  Learning  Programs 


^  G  AO  Joint  Staff’s  ADL  Plan 

Accountability  *  iPrtegnty  *  Relnaftllity 

Program  Successes 

According  to  Joint  Staff  officiais: 

•  Prototype  Portal  established. 

•  Beta  test  of  the  FY  01  Joint  Operations  course  allowed 
Reserve  Component  Officers  to  complete  a  portion  of  the 
JPME  II  education  requirement  through  ADL  utilizing  the 
Internet. 

•  Through  U.S.  Joint  Forces  Command  developed  web-based 
joint  training  to  prepare  for  Joint  Task  Force  Headquarters 
Staff  and  Combatant  Commander  Battle  Staff  exercises. 

•  Established  the  first  Joint  Staff  ADL  program  element  code 
and  included  program  requirements  in  the  fiscal  years  2003 
through  2007  program  objective  memorandum  (POM). 
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Briefing  Section  X:  Joint  Staff’s  Advanced 
Distributed  Learning  Programs 


^  G  AO  Joint  Staff’s  ADL  Plan 

Accountability  *  Integrity  *  Reliability 

Challenges 

Cultural 

•  Services’  unit  commanders  need  trained  and  ready  joint  staff  officers 
upon  arrival. 

•  Course  material  and  lessons  plans  are  not  shared. 

Policy 

•  OSD  policy  that  specifically  addresses  the  use  of  ADL  is  in  an  early 
formulation  stage. 

•  Without  specific  OSD  policy  or  guidance,  the  Joint  Staff  and  services: 

•  have  no  requirement  to  change  or  update  their  regulations  to  reflect  the 
availability  of  or  use  of  new  ADL  technologies;  and 

•  are  hindered  from  developing  coordinated  and  integrated  policy  that  reflects 
guidance  that  promotes  ADL  implementation. 

Financial 

•  Current  projected  funding  does  not  ensure  full  implementation  of  the 
program. 

•  Funding  is  dependent  upon  combatant  commanders’  voluntarily 
reprogramming  funds  to  support  program. 

•  Joint  courseware  funding  and  management  not  systematically 
integrated  into  planning,  programming,  and  budgeting  system. 
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We  reviewed  the  Department  of  Defense’s  (DOD)  Advanced  Distributed 
Learning  (ADL)  programs  to  determine  the  programs’  expectations, 
implementation  status,  and  major  challenges.  We  collected,  reviewed,  and 
analyzed  relevant  program  information  and  conducted  interviews  with 
DOD  officials  responsible  for  distance  learning  programs  and  from  the 
Office  of  the  Deputy  Under  Secretary  of  Defense  for  Readiness  and 
Training;  Advanced  Distributed  Learning  Co-Laboratory,  Alexandria, 
Virginia;  Joint  Advanced  Distributed  Learning  Co-Laboratory,  Orlando, 
Florida;  Department  of  the  Army,  Deputy  Chief  of  Staff  for  Operations — 
Training;  the  Army  Distance  Learning  Program,  U.S.  Army  Training  and 
Doctrine  Command;  U.S.  Army  National  Guard  Bureau,  Distributed 
Training  Technology  Project;  Department  of  the  Navy,  Office  of  the  Chief 
of  Naval  Operations — Education;  U.S.  Naval  Education  and  Training 
Command,  Office  of  Naval  Education  and  Training;  U.S.  Marine  Corps 
Training  and  Education  Command,  Distance  Learning  Center;  Department 
of  the  U.S.  Air  Force,  Office  of  the  Deputy  Chief  of  Staff  for  Personnel, 
Learning  and  Force  Development;  U.S.  Air  Force  Air  Education  and 
Training  Command,  Air  Force  Institute  for  Advanced  Distributed  Learning; 
U.S.  Air  Force  Office  of  Air  Force  Reserve,  Education,  Training,  Readiness 
Policy;  U.S.  Air  National  Guard,  Distributed  Learning  Program;  Office  of 
the  Chairman  of  the  Joint  Chiefs  of  Staff,  Joint  Doctrine  Education  and 
Training  Division;  and  Department  of  Defense,  Defense  Acquisition 
University. 

To  determine  DOD’s  expectations  for  its  programs,  we  reviewed 
executive,  congressional  and  departmental  guidance  related  to  developing 
DOD-wide  ADL  programs.  We  reviewed  and  analyzed  the  Office  of  the 
Secretary  of  Defense’s  (OSD),  the  military  services’,  and  the  Joint  Staffs 
ADL  strategy  and  implementation  plans  and  OSD’s  Training 
Transformation  Plan.  We  interviewed  OSD,  service,  and  Joint  Staff  ADL 
program  personnel  to  obtain  their  views  about  OSD’s  and  their  service-  or 
Joint  Staff-specific  ADL  program  expectations. 

To  determine  the  implementation  status  of  OSD’s,  the  services’,  and  Joint 
Staffs  ADL  programs,  we  provided  OSD,  service,  and  Joint  Staff  ADL 
program  officials  a  detailed  list  of  questions  concerning  program  vision, 
strategy,  implementation  status,  number  of  ADL  courses,  program 
successes,  and  challenges.  We  reviewed  their  written  responses,  if 
provided,  and  followed  up  with  face-to-face  interviews  to  clarify  or  obtain 
additional  information  if  necessary.  We  reviewed,  and  compared  OSD’s, 
the  services’,  and  Joint  Staffs  ADL  strategies  and  implementation  plans. 
We  interviewed  ADL  program  officials  and  collected  other  documents  as 
necessary  to  determine  the  status  of  the  programs  as  compared  to  their 
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ADL  program  implementation  plans.  Additionally,  for  fiscal  years  1999 
through  2002,  we  obtained,  analyzed,  and  compared  information  about  the 
amount  of  funding  OSD,  the  services,  and  the  Joint  Staff  reportedly 
received  for  their  ADL  programs.  For  the  same  fiscal  years,  we  obtained 
and  reviewed  the  amount  of  funding  DOD  and  the  services  received  as 
reported  for  Budget  Activity  3  in  each  of  the  components  Operations  and 
Maintenance  budgets  (BAS  fimds  all  training  and  recruiting  programs)  and 
compared  the  overall  training  budgets  to  the  amount  of  funding  each 
reportedly  allocated  for  ADL  programs.  In  addition,  we  obtained  and 
analyzed  the  amount  of  funding  that  OSD,  the  services,  and  the  Joint  Staff 
reported  that  they  need  and  have  programmed  for  future  ADL 
requirements  for  fiscal  years  2003  through  2007.  We  compared  the 
amounts  reported  as  needed  to  implement  program  plans  with  the 
amounts  included  in  OSD’s,  the  services’,  and  the  Joint  Staff ‘s  program 
objective  memorandums  for  fiscal  years  2003  through  2007.  The  dollar 
amounts  shown  in  this  report  are  as  of  August  31,  2002.  We  did  not 
independently  verify  the  dollar  amounts  reported  in  OSD’s  and  the 
services’  budgets,  nor  did  we  independently  verify  the  amount  of  funding 
OSD,  the  services,  and  the  Joint  Staff  reportedly  allocated  for  their  ADL 
programs. 

To  determine  major  challenges  affecting  OSD’s,  the  services’  and  the  Joint 
Staffs  ADL  program  implementation,  we  provided  ADL  program  officials  a 
detailed  list  of  questions  that  included  specific  questions  related  to 
challenges  ADL  program  managers  face  that  affect  their  ability  to  execute 
programs  that  achieve  their  expectations.  We  reviewed  their  written 
responses,  if  provided,  and  followed  up  with  face-to-face-interviews  to 
clarify  or  obtain  additional  information  as  necessary.  We  did  a 
comparative  analysis  of  the  comments  they  provided.  We  compiled  a  list  of 
challenges  for  OSD,  each  service,  and  the  Joint  Staff.  We  provided  the  lists 
to  each  for  their  review  and  verification.  The  challenges  cited  by  ADL 
officials  were  grouped  into  four  basic  categories.  During  our  exit  briefing, 
we  provided  ADL  representatives  from  OSD,  the  services,  and  the  Joint 
Staff  with  a  summary  of  the  challenges  noted  during  our  review  and  asked 
for  their  comments.  It  was  the  consensus  of  those  ADL  program  officials 
that  the  challenges  we  identified  are  valid. 

We  did  not  assess  the  effectiveness  of  the  programs  at  this  time  because 
most  are  in  the  early  stages  of  implementation. 
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Directives  and  Guidance  for  DOD’s  ADL 
Programs 


1995 


1996 


1997 


1998 


1999 


QDR-  develop  ADL  Co-Laboratory,  Defense 


strategy  to 
modernize 
education  and 
training 

Established-ADL 
Initiative-Office  of 
Science  and 
Technology  Policy 


Alexandria,  Va. 


Authorization 
Act-Develop 
ADL  strategic 
and 

implementation 

plan 

ADL 

Strategic  Plan 

Joint  ADL 
Co-Laboratory, 
Orlando,  Fla. 


2000 

2001 

2002 

Academic  ADL 
Co-labo  ratory, 
Madison,  Wis. 

QDR-Di  rected 
training 
transformation 
as  a  key  to 

Defense  Planning 
Guidance-develop 
a  training  transformation 
strategy 

Released 

SCORM 

Version  1 

DO  D-wide 
transformation 

Strategic  Plan  for 
Transforming 

Training 

ADL 

Implementation 

Plan 

ADL  Partnership 

Lab,  Telford,  England 

1995 


1996 


1997 


1998 


1999 


2000 


2001 


2002 


Established  the 
Army  Distance 
Leaning  Program 

National  Guard  Bureau- 
Distributive  Training 
Technology  Project 
(DTTP) 

Demonstration  DTTP 


The  Army 
Distance  Learning 
Program  (TADLP) 
implementation 

DTTP  expanded 
to  50  states  and 
4  territories. 


Army  University 
Access  Online 


1995 


1996 


1997 


1998 


1999 


2000 


2001 


2002 


1995 


1996 


1997 


1998 


1999 


2000 


2001 


2002 


Marine  Corps  Distance 

Learning  Program  (MLDLP)  established 


Air  Force 


1995 


1996 


1997 


1998 


1999 


2000 


2001 


2002 


Air  Force  Institute  for  Advanced 
Distribution  Learning  (AFIADL)  estabiished 


QDR  =  Quadrennial  Defense  Review 

ADL  =  Advanced  Distributed  Learning 

SCQRM  =  Sharable  Content  Object  Reference  Model 

TADLP  =  The  Army  Distance  Learning  Program 

DTTP  =  Distributive  Training  Technology  Project 

MCDLP  =  Marine  Caps  Distance  Learning  Program 

AFIADL=Air  Force  Institute  for  Advanced  Distributed  Learning 
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Department  of  Defense 


PERSONNEL  AND 
READINESS 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 


4000  DEFENSE  PENTAGON 
WASHINGTON,  D  C.  20301-4000 


February  20,  2003 


Mr.  Neal  P.  Curtin 
Director,  Defense  Capabilities  and 
Management 

US  General  Accounting  Office 
Washington  DC  20548 

Dear  Mr.  Curtin, 

This  is  the  Department  of  Defense  (DoD)  response  to  the  General  Accounting  Office 
(GAO)  draft  report,  GAO-03-393,  "MILITARY  TRANSFORMATION:  Progress  and 
Challenges  for  DOD’s  Advanced  Distributed  Learning  Programs”  dated  January  24, 
2003.  The  Department  concurs  with  the  draft  report  as  presented. 

Your  report  closely  captures  the  present  baseline  of  Advanced  Distributed  Learning 
(ADL)  for  the  Office  of  the  Secretary  of  Defense  and  Services.  While  concurring  with 
the  report,  the  Department  makes  the  following  observations: 

•  ADL  is  an  evolving  program  and  is  a  critical  enabler  for  transforming  DoD 
training. 

•  Beyond  the  successes  identified  in  the  report,  there  are  numerous  excellent  ADL 
projects  that  are  ongoing  within  various  DoD  Components  and  Agencies. 

•  The  ADL  effort  has  been  recognized  by  national  organizations  with  awards  for  its 
leadership  role  in  establishing  a  new  distributed  learning  framework  for 
government,  industry,  and  academia. 

The  Department  appreciates  the  evaluation  team’s  inclusion  of  our  previously 
provided  informal  comments  and  this  opportunity  to  provide  further  comments  on  the 
draft  report.  Technical  comments  were  also  provided  to  the  GAO  for  consideration  in  the 
final  report. 


Sincerely, 

/Lj  i 

Paul  W. 

Deputy  Under  Secretary  of  Defense 
Readiness 


Mayberry 


(350172) 
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GAO’S  Mission 

The  General  Accounting  Office,  the  audit,  evaluation  and  investigative  arm  of 
Congress,  exists  to  support  Congress  in  meeting  its  constitutional  responsibilities 
and  to  help  improve  the  performance  and  accountability  of  the  federal 
government  for  the  American  people.  GAO  examines  the  use  of  public  funds; 
evaluates  federal  programs  and  policies;  and  provides  analyses, 
recommendations,  and  other  assistance  to  help  Congress  make  informed 
oversight,  policy,  and  funding  decisions.  GAO’s  commitment  to  good  government 
is  reflected  in  its  core  values  of  accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost  is 
through  the  Internet.  GAO’s  Web  site  (www.gao.gov)  contains  abstracts  and  full- 
text  files  of  current  reports  and  testimony  and  an  expanding  archive  of  older 
products.  The  Web  site  features  a  search  engine  to  help  you  locate  documents 
using  key  words  and  phrases.  You  can  print  these  documents  in  their  entirety, 
including  charts  and  other  graphics. 

Each  day,  GAO  issues  a  hst  of  newly  released  reports,  testimony,  and 
correspondence.  GAO  posts  this  list,  known  as  “Today’s  Reports,”  on  its  Web  site 
daily.  The  list  contains  links  to  the  full-text  document  files.  To  have  GAO  e-mail 
this  hst  to  you  every  afternoon,  go  to  www.gao.gov  and  select  “Subscribe  to  daily 
E-mail  alert  for  newly  released  products”  under  the  GAO  Reports  heading. 

Order  by  Mail  or  Phone 

The  first  copy  of  each  printed  report  is  free.  Additional  copies  are  $2  each.  A 
check  or  money  order  should  be  made  out  to  the  Superintendent  of  Documents. 
GAO  also  accepts  VISA  and  Mastercard.  Orders  for  100  or  more  copies  mailed  to  a 
single  address  are  discounted  25  percent.  Orders  should  be  sent  to: 

U.S.  General  Accounting  Office 

441  G  Street  NW,  Room  LM 

Washington,  D.C.  20548 

To  order  by  Phone:  Voice:  (202)  512-6000 

TDD:  (202)  512-2537 

Fax:  (202)  512-6061 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Public  Affairs 

Jeff  Nelligan,  managing  director,  NelliganJ@gao.gov  (202)  512-4800 

U.S.  General  Accounting  Office,  441  G  Street  NW,  Room  7149 

Washington,  D.C.  20548 
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